US-China Education Review A 1 (2011) 22-30 

Earlier title: US-China Education Review, ISSN 1548-6613 




The Design and Development of a Context-Rich, Photo-Based 
Online Testing to Assess Students’ Science Learning* 



Min- Jin Lin 

National Dong Hwa University, Hualien City, Taiwan 

Chorng-Jee Guo 

National Changhua University of Education, Changhua City, Taiwan 

Chia-Er Hsu 

National Dong Hwa University, Hualien City, Taiwan 

This study designed and developed a CP-MCT (content-rich, photo-based multiple choice online test) to assess 
whether college students can apply the basic light concept to interpret daily light phenomena. One hundred college 
students volunteered to take the CP-MCT, and the results were statistically analyzed by applying t - test or ANOVA 
(Analysis of Variance). Using the results obtained in this study as working examples, we continue to discuss the 
instructional applications and the educational implications of the CP-MCT. The CP-MCT was shown to overcome 
some shortcomings of the traditional MCT, while maintaining its advantages in terms of cost and time efficiency as 
well as the ease to administer. Moreover, with the usage of a free Website (retrieved from http://www.my3q.com), 
it relieved the burden of school teachers worrying about the techniques of implementing an online testing. The idea 
of CP-MCT can be applied in the pre-service and in-service teacher education to train the teachers in the design and 
development of the CP-MCT to assess their students’ science learning. 
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Introduction 

Research studies on science learning and teaching have been a prominent area in science education 
research in the past few decades (National Research Council, 2007). In the course of science learning, it is easy 
for students to form and hold misconceptions, either prior to or after formal instruction. Furthermore, they are 
tenacious to change and may interfere with formal science teaching and learning. As far as science instruction 
is concerned, it is important for science teachers to detect and identify students’ misconceptions, so that they 
can help students correct the misconceptions in time. There were many assessment methods developed to 
identify students’ misconceptions in science, such as predict, observe and explain (White, 1982); the 
interview-about-instances (Osborne & Gilbert, 1980); the interview-about-events (Scholium & Happs, 1982); 
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concept map (Ruiz-Primo, 2004); concept cartoon (Stephenson & Warwick, n. d.) and two-tier diagnosis test 
(Treagust, 1988) etc.. Other assessment methods involved pencil and paper tests such as multiple-choices or 
those described by Stead and Osborne (1980), Karrqvist and Anderson (1983), and Eaton, Harding, and 
Anderson (1985). Online test has also been used as a way of science assessment. An example prepared by the 
NWEA (Northwest Evaluation Association) can be seen in http://www.nwea.org/assets/documentlibrary/ 
Science_single.pdf. Although these assessment methods were effective in diagnosing scientific misconceptions, 
they seemed to be time consuming and difficult for untrained teachers to prepare and use, so they were not 
popular in the science classrooms yet. 

Hence, many school teachers are still used to applying traditional tests (such as multiple-choices, 
fill-the -blank, short answer or short essay, etc.) to assess the outcomes of students’ science learning. Students 
answer the tests by selecting an item or recall information to write the answers. These tests may be standardized 
by governmental or private agencies or created by school teachers (retrieved from http://jonathan.mueller. 
faculty.noctrl.edu/toolbox/whatisit.htm). Among these formats, MCT (multiple-choices test) was used frequently 
because they were less time consuming and also relatively easy to validate and determine the internal 
consistency (Taylor & Watson, 2000). Many MCT formats were text-based, and were generally accompanied 
with simplified diagrams, sketch of figures, or formula to illustrate abstract science concept or to test students’ 
understanding of scientific concepts. Those formats had some drawbacks. For examples, students may not be 
familiar with the scientific terminology (Wang & Guo, 1992), the formula was too abstract for students to 
understand and make sense (Popov, Zackrisson, & Olofsson, n. d.). Also, they tended to be context-free, without 
the connections to the daily phenomena. Some students who did well in the MCT failed to learn how to interpret 
the daily phenomena by applying basic scientific concepts learned in the science classrooms. 

In short, traditional MCT has its disadvantages. Firstly, the test is focused more on abstract science 
concept which mostly is context-free without a good linkage with the daily life experience. So, students usually 
cannot apply what they have learned in the classroom to interpret daily phenomena. Secondly, test results 
generally do not provide enough information regarding to what kind of misconceptions students might have. 
Thirdly, students who are good at taking tests may make good scores without a true understanding of the 
science concept (Shepard, 1989). Fourthly, the test results are often recorded as scores or ranking to 
differentiate students’ abilities, but teachers move on without making much change in the future instruction 
(retrieved from http://www.encyclopedia.com/doc/lGl-111616029. html). 

Yet, the development of a teaching approach to help students clarify their misconceptions and bring about 
conceptual change was important for teachers to plan for the future instructions (Fetherstonhaugh, Happs, & 
Treagust, 1987; National Research Council, 2007). Bell (2007) also pointed out that the classroom assessment 
of science learning was to help school teachers to understand and enhance students’ learning. It seemed that the 
traditional MCT cannot help teachers pinpoint students’ science misconceptions, and it did not really provide 
enough hints for teachers to enhance students’ science learning. It seemed that the traditional MCT cannot 
fulfill the aim of science classroom assessment, in terms of providing feedbacks to help further learning. 

Thus, the researchers are motivated to twist the traditional MCT somehow to meet three goals. Firstly, we 
want to design and develop a CP-MCT that closely related to the daily phenomena, so school teachers can use it 
to guide students to interpret the phenomena by applying their basic scientific concept. Secondly, the answer 
choices of CP-MCT will correspond to what kinds of misconceptions students hold. Thirdly, it would still be 
easy to administer and be time efficient. 
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To illustrate the main ideas involved in the design of CP-MCT, we chose light concept as the theme to 
make a demonstration. Light is a common natural phenomenon that can be seen in the physical world on a daily 
basis. Light concept is crucial to be learned as part of the requirement for science literacy that was illustrated in 
science for all Americans (Rutherford & Ahlgren, 1990). Most schools taught light concepts since elementary 
schools, but many students at different school levels held alternative conceptions of light regardless of countries 
and cultures. Moreover, these alternative conceptions influenced students’ future science learning 
(Fetherstonhaugh et al., 1987). 

In Taiwan, non-science major college students only learned physics up to junior high school. We wonder 
whether college students still remember the basic light concepts (such as the propagation, reflection, refraction 
of light, the shadow formation, etc.) learned in junior high school and be able to apply these concepts to explain 
the daily phenomena relevant to light regardless of their majors. Therefore, this study designed and developed a 
CP-MCT as an assessment instrument, and investigated some science major and some non-science major 
college students to see if they could apply the basic light concept to explain the daily phenomena associated 
with light. Based on the test developed, the research questions were as follows: 

(1) Can college students interpret the daily phenomena properly by applying basic light concepts? 

(2) Did the background of the sample students make a difference on their test results? 

Literature Review 

Scientific Concept and Students’ Misconception of Light 

In this study, we addressed three light concepts such as light propagation, shadow formation, and the 
scattering of light. The scientific interpretations of these three light concepts were: (1) Light emits from the 
light sources and travels in straight line; (2) When light encountered some opaque objects, it can not come 
through and forms shadows of such objects. Shadows are similar to the shape of objects, and they do not have 
the same quality, color or line/grains as the corresponding objects; and (3) The reflection or scattering of light 
from a rough surface is related to its surface structure and other special features. 

There are abundant literatures regarding to the study of misconceptions of light. Wang and Guo (1992) 
studied the optics concept of eighth and ninth grade students and found the following results: (1) Students used 
“propagation of light along straight path, refraction and projection” to explaine the propagation along straight 
path and the reflection of light, such as shadow, image of formation by pinhole, image formation by mirrors, 
seeing the objects, etc.; (2) The concepts of real image were generally confused with virtual image; and (3) 
Students’ misconceptions about refraction (such as light going from air to water) were very common. Students 
understood that water bends the light but they were unable to specify its direction (retrieved from 
http://dorise.sec.ntnu.edu.tw/JCSE/paper_detail.php ?pid=037 11992000 0030073). Stead and Osborne (1980) 
administered a survey with 144 students (12 years old) who did not study light in the previous 12 months, and 
with 235 students (13 years old) who had recently studied a unit of light. Results indicated that both groups 
held similar views. Most children recognized objects that were sources of light. Some children thought that 
light did not travel at all. Many students figured that light was only present if observable effects were witnessed. 
Some learners thought that seeing was an active eye process with something leaving the eyes. We designed and 
developed the answer choices of our innovative CP-MCT by using the scientific concepts as the correct 
answers and misconceptions as distracting answer choices. 
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Assessment Tools for Analyzing Alternative Conception of Light 

Wang and Guo (1992) designed and used a multiple -choice test for probing junior high school students’ 
misconceptions in optics. Chen, Chang, and Guo (2004) designed and developed a diagnostic instrument to 
investigate students’ alternative conceptions of reflection and refraction of light. Both instruments mentioned 
above were designed carefully based on solid literature reviews. However, they were text-based and with some 
diagrams or sketch of figures to test students’ misconceptions. Stephenson and Warwick (n. d.) used concept 
cartoons to support progression in students’ understanding of light. Huang (2005) designed and developed a 
two-tier diagnosing system to analyze students’ misconception of light. Lau (2006) designed and developed a 
scientific cartoon material to discuss with students about the misconceptions of light shown in the cartoon. 
Those assessment instruments were useful in assessing students’ misconceptions, but they were not designed or 
developed to assess students’ interpretation of the daily phenomena by applying basic light concept. 

Methods 

The rationale of the design and development of the CP-MCT assessment instrument was based on two 
ideas: “They deal with phenomena that are authentically observable” and “They are used to assess students’ 
interpretation of the daily phenomena by applying light concepts”. Each test item is consisted of a question and 
a picture together with a set of answer choices which were made based on the literature reviews that reveal 
students’ misconception of light in certain aspects. Hence, the teachers can identify students’ categories of 
misconception based on the answer choices they made. The idea is similar to the concept cartoon except that 
there were no characters arguing different viewpoints in this photo-based online testing. 

There were totally five CP-MCT questions (see Appendix). Question 1 is a simple picture to assess the 
concept of shadow formation. Question 2 is another example of how sunlight comes through an anti-typhoon 
board of the window to make shadows on the white board inside the room. It also involves the reflection of light 
from a smooth surface. Questions 3 deals with the forming of shadows and how the coarseness of the surface and 
material influence the lightness or darkness of the stone. Questions 4 and 5 are related to the phenomena seen in 
the water surface and sea surface. Students need to understand that since there are ripples and wavelets on the 
water and sea surfaces, the resulting reflection of light is unsteady and not unidirectional. 

The sample included 100 college students who volunteered to answer those questions in a Website 
(retrieved from http://www.my3q.com). This Website was free to use, and allowed users to upload the pictures 
associated with the multiple-choice questions and a set of answer choices. The Website provides an automatic 
feedback to the users by calculating the descriptive statistics and drawing pie charts with regard to the numbers 
of persons on each item chosen. It is very powerful and convenient to use, the users (such as school teachers) 
do not have to worry about the technical side of the online testing. So, it is easy to administer as well as cost 
and be time efficient for school teachers to implement their CP-MCT. The results of this online CP-MCT were 
used to conduct statistical analysis including frequency, t- test, and AN OVA to find out whether there were any 
significant differences among samples that had different backgrounds. 

Results 

Frequency of Answer Choice Made in Each Question 

The frequency and pie charts of the sample’s answers to the questions can be seen in the right-side of each 
CP-MCT as shown in appendix. The concept tested in question 1 was a simple concept about shadow formation of 
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a plant pot under the sun, and the correct rate was 93%. Question 2 was not a familiar example of shadows 
formation as usually shown in the textbook. It required students to apply their basic light concepts regarding to 
“light travel in straight line, and when it encounters some opaque objects, it can not come through and form 
shadows of such objects”. The correct rate was 82%. Question 3 tested on the concept about light reflection on a 
rough and coarse surface of a stone. The correct rate was 87%. Question 4 required the application of “light 
scattering” concept by a rippling water surface to interpret the reason why the tree shadows in the pond were hazy. 
To our surprise, the correct rate was only 61%. Question 5 asked why the sunlight appeared to blink on the sea 
surface during sunrise. Students should understand that it was caused by the scattering of sunlight by the sea 
wavelets, a result indicating that light was reflected in different directions by the wavering sea surface. The correct 
rate of question 5 was 74%. The lower performance in questions 4 and 5 may be due to the unfamiliarity of the 
examples that were not commonly seen in the textbooks, or it may be due to the fact that students did not know 
how to interpret the daily phenomena using the light scattering concept learned previously. 

Differences of the Test Results Due to Students’ Background 

We applied t-test to investigate if “taking freshman physics”, “gender” and “majors” influenced the test 
outcomes. In the data analysis, if a student answered a question correctly, it would be scored as 1 for that 
question. Otherwise, it would be scored 0. From Table 1, it was shown that gender did not make a significant 
difference among the samples. However, “taking freshman physics” made significant differences on questions 
1, 3, 4 and 5, while “majors” made significant differences on questions 1, 3, and 4. The probability value 
(called /7-value and will be abbreviated as p hereafter) of a statistical hypothesis test is equal to the significance 
level of the test for which we would reject the null hypothesis. The /7-value is compared with the actual 
significance level of our test and, if it is smaller, the result is significant. That is, if the null hypothesis were to 
be rejected at the 5% (or 1% or 0.1%) significance level, this would be reported as “p < 0.05”, or “p < 0.01” or 
“p < 0.001” respectively. 



Table 1 



T -Test Regarding to “Taking Freshman Physics ”, “Gender” and “ Majors ” 



Question 


Taking freshman 
physics? Mean 
(standard deviation) 


t 


P 


Gender 

Mean (standard 
deviation) 


t 


Pr 


Science or 
engineering major? 
Mean (standard 
deviation) 


t 


P 




No 


Yes 






Male 


Female 






No 


Yes 








N= 58 


N = 42 






N = 45 


N= 55 






N = 54 


N = 46 






1 


0.86 


1.00 


-3.02 


0.00*** 


0.96 


0.89 


1.23 


0.22 


0.87 


0.98 


-2.12 


0.04* 


(0.35) 


(0.00) 


(0.21) 


(0.31) 


(0.34) 


(0.15) 


2 


0.81 

(0.40) 


0.86 

(0.35) 


-0.61 


0.54 


0.89 

(0.32) 


0.78 

(0.42) 


1.46 


0.15 


0.78 

(0.42) 


0.89 

(0.31) 


-1.54 


0.13 


o 


0.79 


0.95 


-2.52 


0.01** 


0.91 


0.82 


1.37 


0.17 


0.76 


0.98 


-3.50 


0.00*** 


j 


(0.41) 


(0.22) 


(0.29) 


(0.39) 


(0.43) 


(0.15) 


A 


0.47 


0.81 


-3.82 


0.00*** 


0.71 


0.53 


1.91 


0.06 


0.50 


0.74 


-2.52 


0.01** 


4 


(0.50) 


(0.40) 


(0.46) 


(0.50) 


(0.50) 


(0.44) 


5 


0.62 

(0.49) 


0.86 

(0.35) 


-2.80 


0.01** 


0.80 

(0.40) 


0.65 

(0.48) 


1.64 


0.10 


0.70 

(0.46) 


0.74 

(0.44) 


-0.39 


0.70 


Total 


3.55 

(1.11) 


4.48 

(0.67) 


-5.17 


0.00*** 


4.27 

(0.86) 


3.67 

(1.12) 


2.99 


0.00*** 


3.61 

(1.12) 


4.33 

(0.82) 


-3.67 


0.00*** 



Notes . V < 0.05; ** p < 0.01; *** p < 0.001. 



Since the questions only required basic light concept, it did not make significant differences regarding 
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physics credits taken among samples as shown in Table 2. As long as students remember the basic light concept 
about the propagation of light, the formation of shadows, the reflection and refraction of light, and the 
scattering of light, they can do well in this test. 

Table 2 



AN OVA Analysis With Regard to Numbers of Physics Credits Taken 



Item 




Physics credits taken 
Mean (standard deviation) 




ANOVA 


0 credit 
N = 43 


1-3 credits 
N= 24 


4-9 credits 
N= 25 


More than 10 credits 
N= 8 


F( pr) 


1 


0.86(0.35) 


0.92(0.28) 


1.00(0.00) 


1.00(0.00) 


1.66(0.18) 


2 


0.81(0.39) 


0.83(0.38) 


0.80(0.41) 


1.00(0.00) 


0.61(0.61) 


3 


0.81(0.39) 


0.79(0.41) 


0.96(0.20) 


1.00(0.00) 


1.71(0.17) 


4 


0.51(0.51) 


0.67(0.48) 


0.72(0.46) 


0.63(0.52) 


1.11(0.35) 


5 


0.70(0.46) 


0.79(0.41) 


0.64(0.49) 


0.88(0.35) 


0.81(0.49) 


Total 


3.70(1.12) 


4.00(0.98) 


4.12(1.05) 


4.50(0.53) 


1.83(0.15) 



Discussion 



The primary purpose of this study is to show the feasibility of using CP-MCT to diagnose students’ 
misconceptions in science, using light concept as an example. Bearing in mind that the test developed in this 
study is meant for finding if students can interpret the daily phenomena by applying basic light concept. If not, 
what are their misconceptions? Since the participants of this test are college students, it is understandable that 
most students got the answers right on the application of light propagation and shadow formation. However, 
questions regarding light scattering in freshwater and seawater (questions 4 and 5 respectively) seemed harder 
for the college students no matter how many physics credit hours they took before. 

The CP-MCT is easy to administer because of the free Website (Retrieved from http://www.my3q.com). It 
is time efficient because this Website provides the basic descriptive statistics for the users, and it saves school 
teachers a lot of time in grading and getting a quick picture of the test results. It is also a powerful diagnostic 
tool for school teachers to understand what kind of misconception students had because the distracters in each 
test item were made based on a careful literature review. Moreover, it is also handy for school teachers to test if 
students really understand the basic light concept and to interpret the daily phenomena associated with light. 
Many students understand the terminology of physics, and they have the skills to apply formula to solve 
complicate physics questions shown in a text -based test. However, they may not be able to interpret the daily 
phenomena associated with light by applying the basic light concept. 

Besides the current study focused on the light concept, the application of CP-MCT can also be applied in 
other subjects that are consisted of context-rich daily phenomena such as geosciences. For example, teachers 
can take a picture of mudflow after typhoon to assess if students can explain what causes the landslide. 
Moreover, the teachers in fine art may find it feasible to assess students’ understanding of how the artist plays 
with light reflection and refraction in an oil picture. However, there are certainly limits on CP-MCT, because it 
is only good for the daily phenomena that can recorded by a camera (digital or regular). Something that is too 
abstract, too small or too big may not be suitable for this type of assessment instrument. However, there are so 
many daily phenomena associated with scientific concepts which can be done with CP-MCT. It will be still a 
powerful tool for school teachers to use to understand how well their students can apply the basic scientific 
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concept to interpret the daily phenomena. 



Educational Implication 

This study proposed a CP-MCT to overcome some shortcomings of the traditional MCT. With the 
availability of existing research findings on students’ misconceptions, it proves to be a very powerful tool for 
identifying students’ misconceptions in science. Prudent use of CP-MCT by science teachers is expected to lead to 
better science instruction. Of course, the usage of CP-MCT is not limited to any particular subject such as physics. 
It can be easily applied to other subject areas in science or even visual arts. It still keeps the advantages of the 
traditional MCT in terms of cost and time efficiency and the ease to administer. Moreover, with the free Website 
(http://www.my3q.com), it waives the worry about techniques of implementing an online testing. The idea of 
CP-MCT can also be applied in the pre-service and in-service teacher education to train the pre-service and 
in-service science teachers for designing and developing the CP-MCT to assess their students’ science learning. 

We will recommend school teachers to try out the ideas proposed in this study as well as trying to use any 
network resources that can relieve the burdens in teaching and assessment. However, the making of answer 
choices based on a careful survey about the misconception literature reviews may be too much work for school 
teachers to fulfill in their busy schedules. It is best that a national or international organization takes the 
responsibility to handle the preparation of the tests. It would be even better that certain Website (similar to the 
concept of Wikipedia) would attract school teachers at different school levels to share the good works of 
CP-MCT among the Web-based teacher community. 
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Appendix: A Context-Rich, Photo-Based Multiple Choice Test 



Question 1: Sun irradiated on a plant pot, how would a shadow be made? 





Answer Choice 1: (93%) 

Light traveled in a straight direction. Some 
sunlight was blocked by the plant pot, and 
some sunlight went through the slits among 
leaves and made shadows. 



Answer Choice 2: (0%) 

When sunlight illuminated on the leaves, it 
conducted photosynthesis to make shadows. 



Answer Choice 3: (3%) 

The plant pot was not transparent, so it 
projected shadows on the same side as the sun. 



Answer Choice 4: (4%) 

Light traveled in curved path according to the 
shape of the plant pot, and it projected 
shadows. 



Question 2: What happened to the white board inside the room when sunlight came through the anti-typhoon board in the 
window? 





Answer Choice 1: (4%) 

Sunlight made a turn to come through 
anti-typhoon board and illuminated on the 
white board. 



Answer Choice 2: (14%) 

The anti-typhoon board blocked the sunlight 
that cast its shadows on the white board. 



Answer Choice 3: (82%) 

Light travels in a straight line, and the part of 
the whiteboard blocked by the anti-typhoon 
board was darker, while the other part was 
lighter. 



Answer Choice 4: (0%) 

When there was strong sunshine outdoors 
light irradiated into the room and cast its 
shadows on the white board. 



(to be continued) 
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Question 3: What happened when sunlight illuminated on the coarse surface of the stone? 

Answer Choice 1: (2%) 

Light stayed on the stone surface because it 
did not go through, or be reflected or be 
absorbed by the stone. 

* Answer Choice 2: (87%) 

The extent of coarseness of stone surface and 
the angle of illumination influenced the 
darkness of the stone surface. 

Answer Choice 3: (2%) 

A stone was not similar to a mirror that 
reflected the images of objects, so there was 
no light reflection on the stone. 

Answer Choice 4: (2%) 

Some parts of the stone were darker because it 
did not absorb or transform light energy. 
Answer Choice 5: (7%) 

The material of the stone was hard, and the 
surface of the stone was rough and uneven, so 
it did not reflect light well. 

Question 4: Why the tree shadows in the pond were hazy? 

Answer Choice 1: (15%) 

It was due to the refraction of light on the 
water surface. 

* Answer Choice 2: (61%) 

It was caused by the wavelet that scattered the 

light. 

Answer Choice 3: (24%) 

It was so because the angle of incidence was 
not the same as the angle of reflection. 

Answer Choice 4: (0%) 

It was so because the water surface could not 
reflect the sunlight. 

Question 5: Why the sunlight blinked on the sea surface during sunrise? 

Answer Choice 1: (74%) 

The sunlight scattered on the sea surface. Due 
to the differences in reflection angles, the 
sun’s image perceived by human eyes was 
different and looked blinking. 

Answer Choice 2: (5%) 

Strong sunlight made human eyes winked, and 
it looked blinking. 

Answer Choice 3: (0%) 

The components of air and components of 
water interacted to come up with a chemical 
reaction to make sunlight blinking. 

Answer Choice 4: (21%) 

Sunlight illuminated on the sea surface and 
refracted, its image entered into human eyes 

and looked blinking. 

Notes. 1. was added in front of the correct answer; 2. Inside the parenthesis is the percentage of the sample that chose that 






answer. 



